Image processing device, image processing method, and program

ABSTRACT

The image processing device includes an image data input unit for receiving data of moving images and still images; an image grouping unit for classifying the moving images and the still images into groups; an image analyzer for analyzing the moving images and the still images classified by group, and obtaining analysis information of the images, and information on relationship between the moving images and the still images; a frame image extractor for extracting frame images from the moving images according to at least one of the analysis information and the relationship information; a layout determining unit for determining a layout of the still images and the frame images according to at least one of the analysis information and the relationship information; and an image arranging unit for arranging the still images and the frame images according to the layout.

BACKGROUND OF THE INVENTION

The present invention relates to an image processing device, an imageprocessing method and program and more particularly to an imageprocessing method and program wherein still images (referred to below asframe images) are extracted from a moving image in data containing stillimages and moving images according to relationship between the stillimages and the moving image, and wherein the still images and the movingimage are arranged.

As recent digital still cameras acquire enhanced performance, some ofthose on the market are now capable of recording high-resolution movingimages. Accordingly, there are demands for printing or placing in aphotograph album not only still images but frame images cut out from amoving image. Against such background, there have been made variouspropositions as to an easy-to-see layout and list of still images andframe images.

JP 2009-118060 A, for example, describes an image display device wherebya characteristics amount is acquired by extracting still images fromimage data of still images or moving images to determine their positionsin a layout in a display area based on the characteristics amount anddisplay images intended for display obtained by reducing the size ofstill images in the form of a list in the display area. The literaturealso describes that where the image data of moving image contains aplurality of contents, the image data of moving image is divided intoevents or segments to extract still images representing the moving imagefrom noise-free portions thereof and arrange the still images close toeach other.

JP 2009-246887 A describes extracting frame image informationcorresponding to a given number of frame images from each of movingimage files stored in a storage medium, and the arrangement order isdetermined in such a manner that an index image corresponding to theextracted frame image information and an index image corresponding tothe still image file read out from the storage medium can be arrangedaccording to an order satisfying given conditions

JP 2009-246897 A describes a case where a list of images containingindex images extracted from a moving image file and index imagesextracted from a still image file is to be printed, that patentliterature describing correcting the size of the index images so as tocoincide with the reference characteristics amount of the sizeinformation, correcting the colors of the index images so as to coincidewith the reference characteristics amount of the color informationbefore printing, so that the printed list of images has a visual unityamong the index images produced from the moving image file and the stillimage file.

SUMMARY OF THE INVENTION

However, where, when extracting a still image from a moving image at atime a previously designated time has elapsed from the beginning of amoving image is extracted in order to extract a noise-free portion as inJP 2009-118060 A, or where noise in the image data of a moving image isanalyzed for still image extraction, the still images extracted from themoving image are certainly still images representative of the movingimage but are not necessarily images the user desires to print as stillimages or use as material for a photograph album.

Further, as in JP 2009-246887 A and JP 2009-246897 A, where, afterextraction of a moving image and still images, arrangement conditionsare determined and image corrections are made to provide unity in thelayout, images most appropriate for printing may not necessarily beextracted or images contained in a moving image, which, when used, wouldprovide unity without requiring image corrections, actually may notnecessarily be used. Further, displaying images in a mere list may givea monotonous, boring impression as prints, photograph albums, electronicalbums, and the like.

It is an object of the invention to provide an image processing device,an image processing method and program for producing more attractiveimage arrangement data by producing a layout according to relationshipobtained by classifying moving images and still images into groups andanalyzing these images.

In order to attain the object described above, the present inventionprovides an image processing device for making a layout of images,comprising:

an image data input unit for receiving data of moving images and stillimages,

an image grouping unit for classifying the moving images and the stillimages into groups under a given condition,

an image analyzer for analyzing the moving images and the still imagesas classified by the group, and obtaining analysis information obtainedby analyzing the moving images and the still images and relationshipinformation indicating relationship between the moving images and thestill images,

a frame image extractor for determining a number of frame images to beextracted from each of the moving images by the group according to atleast one of the analysis information and the relationship information,and extracting a determined number of frame images,

a layout determining unit for determining a layout of the still imagesand the frame images by the group according to at least one of theanalysis information and the relationship information, and

an image arranging unit for arranging the still images and the frameimages by the group according to the layout.

Also, the present invention provides an image processing method ofproducing a layout of images, comprising:

an image data entering step of receiving data of moving image and stillimage,

an image grouping step of classifying the moving images and the stillimage into groups under a given condition,

an image analyzing step of analyzing the moving images and the stillimages as classified by the group, and obtaining analysis informationobtained by analyzing the moving images and the still images andrelationship information indicating relationship between the movingimages and the still images,

a frame image extracting step of determining a number of frame images tobe extracted from each of the moving images by the group according to atleast one of the analysis information and the relationship information,and extracting a determined number of frame images,

a layout determining step of determining a layout of the still imagesand the frame images by the group according to at least one of theanalysis information and the relationship information, and

an image arranging step of arranging the still images and the frameimages by the group according to the layout.

Also, the present invention provides a non-transitory computer readablemedium storing a computer program for causing a computer to perform animage processing method of making a layout of images described above.

The present invention enables production of arrangement data where astill image and optimal frame images are arranged according to therelationship between a moving image and still images without requiringmuch effort for selecting frame images the user desires to use from animage even when the material data contain moving image data and stillimage data mixed therein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to FIG. 1C are views for explaining an idea of classificationand frame image extraction according to the present invention.

FIG. 2 is a block diagram showing an example of a configuration of theimage processing device for implementing the image processing method ofthe present invention.

FIG. 3 is a flowchart indicating an example of the flow of the imageprocessing method of the present invention.

FIG. 4A is a view for explaining an example of classification; FIG. 4Bis a view for explaining an example of classification where a frameimage extraction was implemented.

FIG. 5 is a view for explaining an example of an effective layoutrelated to the same subject.

FIG. 6 is a view for explaining an example containing both still imagesand moving images and where the still images and moving images arehighly related.

FIG. 7 is a view for explaining an example where frame images have beenextracted from the moving images.

FIG. 8 is a view for explaining an example of layout containing bothstill images and moving images and where the still images and movingimages are highly related.

FIG. 9 is a view for explaining another example of layout containingboth still images and moving images and where the still images andmoving images are highly related.

FIGS. 10A and 10B are views each for explaining an example of layoutcontaining both still images and moving images and where the stillimages and moving images are related to a lower degree.

FIGS. 11A to 11C are views each for explaining an example of layoutwhere only frame images extracted from a moving image are arranged.

FIG. 12A is a view for explaining an example showing a state beforeclassification is effected; FIG. 12B is a view for explaining an exampleshowing a state after classification is effected; FIG. 12C is a view forexplaining an example where images have been arranged; FIG. 12D is aview for explaining an example illustrating an example of template forrecomposition of a page for which a recomposition instruction has beenentered.

FIG. 13 is a view for explaining an example where recommended images areextracted from moving images and still images classified into events.

FIG. 14A is a view for explaining an example where classification iseffected by similar scene and a recommended image is determined andextracted. FIG. 14B is a view for explaining an example whereclassification is effected with respect to designated person and arecommended image is determined and extracted.

FIG. 15 is a view for explaining an example of layout before a face anda vanishing point are detected.

FIG. 16 is a view for explaining an example of layout where a layout iseffected using a face and a vanishing point.

FIGS. 17A and 17B are views for explaining an example where frame imagesshowing a change in face direction are extracted and arranged in a givendirection.

FIG. 18 is a view for explaining an example of layout where anotherimage is superposed so as not to hide a major subject.

FIG. 19 is a view for explaining an example of layout where a givenspace located in a direction of a subject's face is secured free ofother images.

FIG. 20A is a view for explaining an example of layout where the facesof target persons are recognized but their eyes are not directed towardthe camera; FIG. 20B is a view for explaining an example of layout wheretheir eyes are directed toward the camera and smiles are observed.

FIG. 21 is a view for explaining an example where a recommended image isautomatically extracted from a moving image.

FIG. 22 is a view for explaining an example of layout for shuffle print.

FIG. 23 is a view for explaining an example of layout where best shotsare arranged chronologically to express a movement.

FIG. 24 is a view for explaining an example where a high-resolutionframe image is produced from a plurality of frame images.

FIG. 25 is a view for explaining an example where images showing a givenperson are extracted from a group of numerous images.

FIG. 26 is a view for explaining an example of registration andclassification processing.

FIG. 27 is a view for explaining another example of registration andclassification processing.

FIG. 28 is a view for explaining an example of layout in a photographalbum.

FIG. 29 is a view for explaining another example of layout in aphotograph album.

FIG. 30 is a view for explaining an example where characters are addedin marginal areas of a photograph album.

FIG. 31 is a view for explaining an example where characters are addedin a vacant region and a region other than a face region of a photographalbum.

FIG. 32 is a view for explaining an example where a background templateis selected according to color information of selected images.

FIG. 33 is a view for explaining an example of composition of a templatefor arranging four images in one page.

FIG. 34 is a view for explaining an example showing how groups in whichthe total number of images exceeds a reference number of images are eachdivided into two or more groups.

FIG. 35 is a view for explaining an example of a screen displaying aresult of classification of images into groups and a line (border line)indicating a reference number of images.

FIG. 36 is a view for explaining an example of images classified into afirst group to a fifth group.

FIG. 37 is a view for explaining an example how frame images have beenextracted from moving images only considering a reference number ofimages.

FIGS. 38A and 38B respectively illustrate how the number of frame imagesis increased according to the number of pages of a photograph albumdesignated by the user, and how the images shown in FIG. 38A are dividedand allocated to 7 pages.

FIG. 39 is a view for explaining an example of layout where a singleframe image is enlarged and positioned at the center of a page.

FIG. 40 is a view for explaining an example showing how the number offrame images is increased to four, and the display size of each of theimages is reduced.

FIGS. 41A to 41C respectively illustrate a moving image containing achild and a rabbit, how a given region containing the face of the childis extracted as a frame image before the face moves and after the facemoves.

FIG. 42A illustrates a still image showing a child and a rabbit. FIG.42B is a view for explaining an example showing how given regionsrespectively containing the child and the rabbit, both belonging to thesame group, are extracted as frame images from a moving image containingthe child and the rabbit.

DETAILED DESCRIPTION OF THE INVENTION

The following describes in detail the image processing device of thepresent invention for implementing the image processing method of theinvention based upon the preferred embodiments illustrated in theaccompanying drawings.

First, a method of the invention for classifying moving images and stillimages and extracting frame images from moving images will be describedby referring to FIG. 1.

FIG. 1 is a view for explaining a concept of classification and frameimage extraction according to the invention. First, when image datacontaining moving images and still images is entered as illustrated inFIG. 1A, classification of moving images and still images is made byevent or by shooting time as illustrated in FIG. 1B. Then, moving imagesand still images of each group are analyzed by group to obtain analysisinformation obtained by analyzing the moving images and still images andinformation on the relationship between the moving images and stillimages. Subsequently, the number of frame images to be extracted fromeach moving image is determined according to the analysis informationand/or the relationship information to extract a determined number offrame images. In an example illustrated in FIG. 1C, for example, threeframe images are extracted from the moving image classified into a firstgroup; two frame images are extracted from the moving image classifiedinto a second group. Thus, appropriate frame images are extracted byextracting frame images according to the analysis information and/or therelationship information.

FIG. 2 is a block diagram illustrating an embodiment of configuration ofan image processing device of the invention for implementing an imageprocessing method of the invention.

An image processing device 10 illustrated in FIG. 2 comprises an imagedata input unit 12, an image display 14, an operating unit 16, an imagerecorder 18, an image grouping unit 19, an image processor 20, a layoutdetermining unit 26, and an image arranging unit 28, all connected by abus 30. The image processor 20 comprises an image analyzer 22 and aframe image extractor 24. In this embodiment, an example for producing aphotograph album will be described.

The image data input unit 12 has an external interface and receivesmoving image data and still image data (moving image/still image data)to be used for producing a photograph album. The image data supplied tothe image data input unit 12 is not specifically limited and may bevarious image data including moving image/still image data acquired bythe user with a digital still camera or a digital video camera andmoving image/still image data downloaded from a network. The externalinterfaces include USB (Universal Serial Bus), memory cardreader/writer, optical discs, and wired and wireless LAN (Local AreaNetwork). Image data supplied to the image data input unit 12 isoutputted as material image data (moving image/still image data).

The image display 14 displays an editing screen of an image processingapplication that runs in the image processing device 10, data whoseimages have been arranged (e.g., arrangement image in a photographalbum), operation screens for the user to perform various operations,and the like.

The operating unit 16 is provided to perform various operations of theimage processing device 10. The operating unit 16 is not particularlylimited, and any of various known operating devices may be used such asa keyboard, a mouse, and a touch panel.

The image recorder 18 records entered material image data (movingimage/still image data), frame image data, layout data of still imagesand frame images, arrangement data representing an arrangement image,and the like. The image recorder 18 may be a semiconductor memory suchas a DRAM (Dynamic Random Access Memory) and a flash memory or amagnetic recording medium such as an HDD (Hard Disk Drive).

The image grouping unit 19 classifies entered material image data(moving image/still image data) into groups according to a givencondition (event). The given condition may be, for example, shootingtime, shooting location obtained by GPS (Global Positioning System),etc., direction in which the subject faces, or movement of the subject.The classification results may be added to material image data asclassification information and may be managed using a table separatelyprovided in the image recorder 18.

The image analyzer 22 comprises a still image analyzer 22 a and a movingimage analyzer 22 b. The image analyzer 22 makes analysis to determinewhether each group contains moving images and still images, that is,whether moving images and still images are mixedly contained.

The still image analyzer 22 a analyzes the still image data for each ofthe groups into which classification has been made by the image groupingunit 19 to obtain a maximum number of frame images that can be extractedfrom a moving image (referred to below as maximum extractable number),analysis information obtained by analyzing a still image, andinformation on relationship between a moving image and a still image(referred to below as relationship information). The maximum extractablenumber may be, for example, a number obtained by subtracting the numberof still images from a predetermined reference number of imagescontained in the same group.

The reference number of images contained in the same group is a maximumnumber of images that may be arranged by group. When, for example, eachgroup of images is assigned one page of a photograph album, thereference number of images is a maximum number of images that may bearranged in one page of the photograph album.

When, for example, a template of a layout for arranging images in presetpositions in a page (predetermined candidate layout) is used, thereference number of images is a maximum number of images that may beused (arranged) in that template. When, for example, a template shown inFIG. 33 is used, four images may be arranged and therefore the referencenumber of images is four. When a template is used, the reference numberof images varies with a template used.

On the other hand, when images are arranged in arbitrary positions in apage without using an existing template (free layout), the imageprocessing device has therein set a maximum number of images that may bearranged in one page considering ease of viewing. In the case of freelayout, therefore, the reference number of images is “a maximum numberof images that may be arranged in one page” as preset in the imageprocessing device 10.

The analysis information may be at least one of, for example, subject'sface recognition result, face direction, the number of subjects, size ofthe subject, position of the subject, direction in which the subjectfaces, subject's facial expression recognition, shooting time, shootinglocation, composition, vanishing point, color, brightness of the wholeimage, vividness of the whole image, identification of a person,shooting condition, resolution of an image, aspect ratio of a image, andshooting mode.

As for the relationship information, representative values thereof maybe obtained by weighting the above information obtained by comparinganalysis information on moving images and still images, therepresentative values being in a range such that an upper limit is 100and a lower limit is 0. Alternatively, the analysis information may beused as relationship information. The composition is exemplified by alandscape showing a person, a building, a sky, and the like; theshooting mode is exemplified by a normal shooting mode, a panoramicshooting mode, a close-up mode, and a person shooting mode.

Similarly to the still image analyzer 22 a, the moving image analyzer 22b analyzes moving image data for each of the groups into whichclassification has been made by the image grouping unit 19 to obtainanalysis information and relationship information. The analysisinformation may also be movement detection information and cameramovement amount (pan, tilt).

The frame image extractor 24 determines, by group, the number of imagesto be extracted from moving images that does not exceed the maximumextractable number according to the analysis information and/orrelationship information of moving images and of still images obtainedby the image analyzer 22 and extracts a determined number of frameimages. For example, when frame images are extracted based on theshooting time, a table as shown in Table 1 is previously produced, andthe extraction number is determined based on that table to extract frameimages from moving images. The data of the extracted frame images (frameimage data) is stored in the image recorder 18.

TABLE 1 Moving image shooting time Number of extracted images 1 s-5 s 1 6 s-10 s 2 . . . . . .

When, for example, the “face direction” is used, the variation in facedirection may be held as fixed value in lieu of using the table. Forexample, when the face direction changed by 15 degrees or more from agiven position, the frame image may be extracted. Further, when theframe image is extracted based on the shooting mode, if a still image isa close-up image, a frame image close to a close-up image may beextracted from a moving image.

The frame image extractor 24 may extract frame images from moving imagesaccording to imaging time (reproduction time) in the analysisinformation on moving images, imaged content in the analysis informationon moving images, and/or relationship information without consideringthe reference number of images.

When the total number of still images and frame images in a groupexceeds the reference number of images, recommended frame images (frameimages judged by the frame image extractor 24 to be preferable) areextracted from the extracted frame images according to the analysisinformation on still images (including extracted from images) withoutexceeding the reference number of images. In this case, frame images maybe each rated with marks in such a manner that, for example, an imageshowing a person's face in a great proportion is given high marks, andframe images given high marks may be used without exceeding thereference number of images.

Alternatively, a group in which the total number of images exceeds thereference number of images may be divided into two or more groups.Suppose that after frame images are extracted by group from movingimages, the total number of still images and frame images exceed four,the reference number of images, in the first to the third groups asillustrated in FIG. 34. Note that the frame images are numbered in FIG.34. The first group contains a total of eight images; the second groupcontains a total of five images; the third group contains a total of siximages. In this case, the fifth and other still and frame images tofollow in each of the first to the third groups are classified intoother groups.

Alternatively, the user may determine a selectable number of frameimages that may be extracted from each moving image considering thereference number of images. As illustrated in FIG. 35, the screendisplays a result of classification of images into groups and a line(border line) indicating the reference number of images. For example,since the third group contains a single moving image, a maximum of fourframe images may be extracted. Accordingly, the user may select a numberof frame images to be extracted from a range of one to four.

Alternatively, when the user does not consider the reference number ofimages so that the extractable number of frame images may be selectedfrom five or more and the total number of still images and frame imagesin a group exceed the reference number, the group may be divided intotwo or more groups.

The frame image extractor 24 may determine the number of frame imagesthat may be extracted from each moving image according to the number ofpages designated by the user.

Suppose, for example, that the user designates production of aphotograph album containing seven pages using images classified into thefirst to the fifth groups as illustrated in FIG. 36. Suppose also thatthe reference number of images is four. Then, when simply the referencenumber of images is only considered, the number of pages of thephotograph album is five because, as illustrated in FIG. 37, one frameimage is extracted from the moving image of the first group, and twoframe images are extracted from the moving image of the second group.

Therefore, the numbers of frame images extracted from each moving imageis increased according to the number of pages designated by the user, sothat, for example, two frame images are extracted from the moving imageof the first group, and five frame images are extracted from the movingimage of the second group as illustrated in FIG. 38A. Then, asillustrated in FIG. 38B, the first group is divided into a groupcontaining two frame images and another group containing three stillimages, while the second group is divided into a group containing twostill images and two frame images and another group containing threeframe images. Thus, the photograph album has seven pages as designatedby the user.

The number of frame images to be extracted from each moving image andhow a group containing a number of images exceeding the reference numberof images is divided may be determined in any manner as appropriatewithin an allowable range according to the number of pages designated bythe user and the reference number of images.

The layout determining unit 26 determines the layout of frame images andstill images by group based on the analysis information and/orrelationship information and produces and outputs layout data. Forexample, when a photograph album is to be produced, each group isassigned one page, and the layout of the page is determined based on theanalysis information and/or relationship information. Layout data isstored in the image recorder 18.

The image arranging unit 28 reads the frame image data and still imagedata from the image recorder 18 based on the layout data, arranges theimages, and produces and outputs the arrangement data. The arrangementdata is stored in the image recorder 18.

Next, the operation of the image processing device 10 of the inventionimplementing the image processing method of the invention will bedescribed.

FIG. 3 is a flowchart indicating an example of the flow of the imageprocessing method of the present invention.

First, data (moving image/still image data) of material moving imagesand still images is entered from the image data input unit 12 (stepS10). The entered moving image/still image data is recorded once in theimage recorder 18 as material image data.

The material image data (moving image/still image data) is read by theimage grouping unit 19 from the image recorder 18 and classified intogroups according to a given condition (event) (step S12).

For example, as illustrated in FIG. 4A, where images are classified byshooting date, one moving image and six still images shot from 10:00 to12:00 on February 11th are classified into the first group; two stillimages and one moving image shot from 12:00 to 15:00 on the same day areclassified into the first group; one moving image shot from 15:00 to17:00 on the same day is classified into the third group; two stillimages shot from 17:00 to 24:00 on the same day are classified into thefourth group; and three still images shot on the following day areclassified into the fifth group.

Next, the moving image/still image data classified into groups areanalyzed by group by the image analyzer 22 to determine whether movingimages and still images are mixed therein (step S14). Processing isperformed in manners specific to cases where moving images and stillimages are mixed (“mixed” in the step S14), cases where only movingimages are contained (“moving images only” in the step S14), and caseswhere only still images are contained (“still images only” in the stepS14), respectively.

In the example illustrated in FIG. 4A, moving images and still imagesare mixedly contained in the first and the second groups (“mixed” in thestep S14); only moving images are contained in the third group (“movingimages only” in the step S14); and only still images are contained inthe fourth and fifth groups (“still images only” in the step S14).Processing is separately performed for the respective groupsaccordingly.

Where moving images and still images are mixedly contained (“mixed” inthe step S14), the still image analyzer 22 a analyzes the still imagedata contained in the group to obtain still image analysis information,so that the maximum number of frame images that can be extracted from amoving image is calculated.

Next, the moving image analyzer 22 b analyzes the moving image datacontained in the group to obtain analysis information on moving images.The still image analyzer 22 a and/or the moving image analyzer 22 bcompares their analysis information to obtain relationship information.Further, the representative values of the respective relationshipinformation are calculated by weighting the analysis information.

Subsequently, the frame image extractor 24 determines the number offrame images to be extracted from each moving image without exceedingthe maximum extractable number of frame images based on the analysisinformation and/or the relationship information and extracts adetermined number of frame images. The still images and the movingimages contained in the same group (event) highly likely represent thesame scene and, therefore, frame images as different in direction of thesubject, scene, composition, etc. from the still images as possible arepreferably extracted.

For example, in the example illustrated in FIG. 4B, with a preconditionthat the maximum number of images of the first group is 8, the stillimage data of the six still images of the first group is analyzed, and,first, the maximum number of frame images is calculated as 2. Where noneof the six images shows the front of a subject, obtained are analysisinformation that there are no images representing the front of a subjectand relationship information that images representing the front of asubject are lacking.

Then, according to the relationship information that images representingfront images of a subject is lacking, frame images representing frontimages of the subject are extracted in a number that is 2 in maximum,i.e., the maximum extractable number of frame images depending on thelength of the moving image (step S16). Thus, when the moving image islong, the same number of frame images as the maximum extractable numberare extracted but when the moving image is short in length as in thecase of the second group shown in FIG. 4B, only three images areextracted within the maximum extractable number according to theanalysis information and relationship information (e.g., number ofscenes, number of detected movements), and, therefore, in some casesonly five images in total including still images may be extracted. Theanalysis information and the relationship information on moving imagesmay be carried over in the extracted frame images.

One may extract frame images representing same scene (composition) andthe same subject but showing the subject face or body directed inslightly different directions such that when these images are arranged,they will show a movement as illustrated in FIG. 5. Such arrangementshowing a movement, where possible, is an effective layout.

In another example, as in the case of the second group illustrated inFIG. 6, when a still image 40 shows a person A and a moving image 42shows the person A, a frame image showing the person A differently thanis shown in the still image 40 may be preferentially extracted from themoving image 42. Alternatively, frame images considering the shootinglocation (shooting place) in lieu of images showing the person A may beextracted using location information such as information obtained usingGPS and the like. Further, frame images showing scenes different incomposition, colors, etc. from the still image 40 may be extracted.

Alternatively, frame images may be extracted considering the resolutionof images.

When only one frame image is extracted from one moving image containedin a group such as the third group shown in FIG. 6, for example, aresulting layout is one as illustrated in FIG. 39, where one frame imageis enlarged and positioned in the center of a page. In this case, noproblem is posed when the frame image extracted from the moving imagehas a high resolution. When the frame image is one extracted from alow-resolution moving image acquired using, for example, a camerafunction provided in a portable telephone, a printout produced therefromwill have a poor image quality. Therefore, when a moving image has aresolution lower than a given resolution and the total number of stillimages and frame images contained in a group is smaller than a givennumber, an adjustment is preferably made by increasing the number offrame images to be extracted from the moving image, so that the displaysize of one image is smaller than a given size in consideration of theimage quality of printouts. When the number of frame images is increasedto four, for example, as illustrated in FIG. 40, the size of each imageis reduced, which necessarily improves the image quality of theresulting printouts.

Alternatively, frame images may be extracted considering the number andsize of subject persons.

When, for example, a moving image contains many persons or the size ofpersons contained in a moving image is small even when only a fewpersons are contained in the moving image, the display size of eachimage is small if a large number of frame images are extracted from themoving image, making it difficult to recognize the persons. When thesize of persons contained in a moving image is smaller than a givensize, adjustment is preferably made by reducing the number of frameimages extracted from the moving image so that the persons are displayedin sizes not smaller than a given size for easy recognition of thepersons.

Alternatively, frame images may be extracted considering the aspectratio of images.

The aspect ratio of a moving image varies depending on, for example, theway a camera is held (vertically or horizontally) and the shooting mode(full high vision 1920×1080; VGA 640×480). Accordingly, it is preferablethat, for example, the number of frame images to be extracted from amoving image having an aspect ratio required for a layout of images isincreased while the number of frame images to be extracted from othermoving images is reduced according to the conditions for producing aphotograph album or an electronic album. When, for example, manyportrait-aspect frame images are required, the number of frame images tobe extracted from portrait-aspect moving images is increased, while thenumber of frame images to be extracted from landscape-aspect movingimages is reduced.

Alternatively, in lieu of extracting frame images in size correspondingto the whole region of a moving image as displayed on the screen, aregion or a part of a moving image as displayed on the screen with whichframe images are to be extracted may be determined, and frame images maybe extracted with a determined region.

When, for example, a moving image contains a child and a rabbit, and thechild's face makes a particularly great movement (i.e., the direction ofthe face changes greatly) as illustrated in FIG. 41A, a given regionincluding the child's face may be determined as region with which imagesare extracted from the moving image so that a given region surrounded bya dotted line in FIG. 41B including the child's face before the facemoves is extracted as a frame image 1 and a given region surrounded by adotted line in FIG. 41C including the child's face after the face movesas time passes is extracted as a frame image 2.

When, for example, a still image in the same group contains the childand the rabbit as illustrated in FIG. 42A and the moving image in thesame group also contains the child and the rabbit as illustrated in FIG.42B, given regions each containing the faces of the child and the rabbitmay be determined as regions for extracting images from the moving imageto extract the given region including the child's face and the givenregion including the rabbit's face each surrounded by a dotted line inFIG. 42B as frame images.

When extraction of frame images from a moving image in a group iscompleted, the representative values of the relationship information ofthe moving image and the frame images are compared with therepresentative value of the relationship information of the still images(step S18). When there is a close relationship between relationshipinformation of the moving image and the frame image and relationshipinformation of the still images (for example, the difference betweentheir representative values is less than plus or minus 10)(“relationship exits” in the step S18), the layout is determinedconsidering the relationship, and the layout data is outputted. Thelayout data is recorded in the image recorder 18 and entered in theimage arranging unit 28, where the layout data is combined with frameimages and still images to produce and output arrangement data (stepS20). The outputted arrangement data is recorded in the image recorder18.

In the second group illustrated in FIG. 7 given by way of example, therelationship information of a still image 44 and frame images 46, 48,and 50 are compared and when it was determined that there is a closerelationship between the still image 44 and the frame images 46, 48, and50, a layout may be made as illustrated in FIG. 8 where the frame images46, 48, and 50 may be arranged beneath a large still image 44.

In another example of layout, a layout may be such that frame images 58are arranged chronologically around a still image 56, as in a layout 54illustrated in FIG. 9.

When, in the step S18, the relationship between relationship informationof a moving image and frame images and relationship information of astill image is not close (for example, the difference between theirrepresentative values is greater than plus or minus 10) (“norelationship” in the step S18), layout data is outputted such that thestill image 44 and the frame images 46, 48, and 50 are simply arrangedwithout considering the relationship information as illustrated in FIG.10A, or a layout 64 where the still image 44 is placed in one page whilethe frame images 46, 48, and 50 are arranged in another page as in alayout 66 as illustrated in FIG. 10B.

The layout data is recorded in the image recorder 18 and entered in theimage arranging unit 28, where the layout data is combined with frameimages and still image to produce and output arrangement data (stepS22). The outputted arrangement data is recorded in the image recorder18.

In cases where judgment “moving image only” is made in the step S14, themoving image analyzer 22 b analyzes the moving image data to obtain theanalysis information. The frame image extractor 24 extracts optimalframe images in a number not exceeding a maximum extractable numberpredetermined for the first group based on the analysis information(step S24).

The third group illustrated in FIG. 4A, for example, contains only onemoving image, and the moving image analyzer 22 b analyzes the movingimage data to obtain the analysis information. Now, when, for example,the user enters an instruction that one frame image be extracted for onescene, and supposing that the third group contains a moving image thatcan be divided into six scenes by composition, one frame for each sceneis extracted. Thus, six frame images are extracted as illustrated inFIG. 4B.

The frame images that are extracted, for example, at given timeintervals, are arranged into a layout 68 where frame images 70 arearranged evenly as illustrated in FIG. 11A and the corresponding layoutdata is outputted. When a moving image contains a close-up portion, alayout 72 may be made where frame images 74, one showing a zoomed-inimage and another a zoomed-out image that follows, are arranged so as tooverlap each other as illustrated in FIG. 11B. When extracted frameimages successively show a child or like subject, extracted framedimages 78 may be arranged into a layout 76 where they are arranged in agiven direction.

As in the case where the “no relationship” judgment is made in the stepS18, the layout data is recorded in the image recorder 18 and entered inthe image arranging unit 28, where the layout data is combined withframe images to produce and output arrangement data (step S22). Theoutputted arrangement data is recorded in the image recorder 18.

When the “still image only” judgment is made in the step S14, the stillimage analyzer 22 a analyzes the still image data to obtain the analysisinformation. The layout determining unit 26 produces and outputs thestill image layout data based on the still image analysis information.The layout data is recorded in the image recorder 18 and entered in theimage arranging unit 28, where the layout data is combined with stillimage to produce and output arrangement data (step S26). The outputtedarrangement data is recorded in the image recorder 18.

Next, referring to FIG. 12, we will describe a case where thearrangement data is recomposed according to user instruction after it isonce produced.

There may be a case where the user desires to recompose some pages ofthe arrangement data of, for example, a photograph album once producedby the image processing method of the present invention. Now, we willconsider a case where, for example, image data of a group of imagesillustrated in FIG. 12A is classified into groups illustrated in FIG.12B to produce the arrangement data of a photograph album illustrated inFIG. 12C by the image processing method of the invention. Suppose thatthe user desires to add more frame images from a moving image in asecond page 80, the user operates the operating unit 19 to giverecomposition instruction for recomposing the second page 80. Thus, theoperating unit 16 outputs recomposition instruction.

The recomposition instruction is entered in the layout determining unit26 and, for example, a plurality of layout candidates (templates 82, 84)are displayed on the image display 14, from which a layout is selectedby the user, and an instruction for re-extracting frame images that suitthe selected layout is outputted.

The re-extraction instruction is entered in the frame image extractor24, and frame images that suit the selected new layout are extracted,registered in the image register 18 and entered in the image arrangingunit 28 to produce and output arrangement data for a recomposed page(second page 80). The outputted arrangement data is recorded in theimage recorder 18.

Recomposition of a page may be effected not only by template selectionby the user but in such a manner that the user designates the number offrame images to be extracted from a moving image so that templateselection is automatically made according to the number of frame images.

The arrangement data produced by the image processing device 10 isprinted by a printer, not shown, and printed pages are booked by abookbinding device, not shown, to produce a photograph album.

EXAMPLES

Next, the image processing method of the invention will be describedreferring to specific examples.

Example 1

FIGS. 13 and 14 illustrate examples where recommended images areextracted by event (group).

First, as illustrated in FIG. 13, moving images and still images areclassified into groups (event A, event B, and event C). In this case,frame images are extracted from a moving image according to the analysisinformation and/or the relationship information.

Next, recommended images are extracted from among images classified intoevents according to the analysis information such as, for example, thenumber of images in an event, the number of subject persons, the size ofa subject, the brightness of the whole image, and vividness of the wholeimage.

In an example of extraction of recommended images, classification ismade by scene showing persons and recommended images are selectedaccording to the position of faces of persons (whether directed towardthe center of the image) and the size of faces (in proportion to thewhole image, etc.) as illustrated in FIG. 14A.

Further, as illustrated in FIG. 14B, extraction may be made in such amanner that classification is effected according to a designated personand the recommended images are determined according to position of aperson's face, size of his/her face, facial expression such as smile,and brightness (luminance).

The extracted recommended images (frame images or a still image) arearranged according to the analysis information and/or the relationshipwith other extracted recommended images, i.e., according to therespective analysis information and/or relationship information.

Example 2

FIGS. 15 to 19 illustrate examples where layout is effected usingdetection of faces and vanishing points.

As illustrated in FIG. 15, for example, a still image is selected fromamong images classified into groups and placed at the center of twospread opposite pages of a photograph album while frame images arearranged around the still image. The layout shown in FIG. 15, however,does not consider the contents of images, so that face directions 90 andvanishing points 92 in the respective images are inconsistent and thusthe layout is inconsistent as a whole.

When a layout is made using the subjects' face directions, the vanishingpoints, and the relationship information among the analysis informationof the still image and the frame images, the resulting layout has adepth at the center and images are placed around the central imageconsidering the directionality of the images based on the vanishingpoints 92 and face directions 94 as illustrated in FIG. 16. In suchlayout, images are arranged with a good balance.

In other examples, faces are detected from a moving image as illustratedin FIGS. 17A and 17B, and the face directions in the analysisinformation and the relationship information are used to extract frameimages showing variation in face direction and arrange the extractedimages in a given direction. In such layout, the images are arranged soas to show the variation in subject's face direction.

In still another example, a major subject 100 in a group is detected asillustrated in FIG. 18, and the positions of the major subject in theanalysis information and the relationship information are used tosuperpose other images so as not to hide the major subject 100. In theresulting layout, the major subject 100 is not hidden and the images aresuperposed effectively.

Further, as illustrated in FIG. 19, a layout may be made so that otherimages are not placed in a space 102 lying in a subject's face direction103 using the subject's face direction in the analysis information andthe relationship information. In the resulting layout, no images areplaced close to and in the direction of the subject's face and thus theimage does not give an oppressive feeling.

Example 3

Examples are shown in FIGS. 20 to 24 where frame images are extractedfrom a moving image. The examples illustrated in FIGS. 20 to 24 showcases where most desirable frame images are extracted. Unless otherwisedescribed, after frame image extraction, layout is effected and frameimages are arranged similarly to the cases of Examples 1 and 2.

When faces 104 of target persons are recognized but their eyes are notdirected toward the camera as illustrated in FIG. 20A, they are not mostdesirable. Therefore, the moving image is analyzed to extract therefroma frame image showing their smiles 108, 110 with their eyes directedtoward the camera as illustrated in FIG. 20B as a most desirable image.

In other examples, recommended frame images are automatically extractedfrom a moving image (most desirable images are extracted) as illustratedin FIG. 21, most desirable images extracted from a moving image arerandomly arranged in a shuffle print layout as illustrated in FIG. 22,and most desirable images are chronologically arranged to show amovement as illustrated in FIG. 23.

Further, as illustrated in FIG. 24, frame images, each shot at time A,B, and C, are extracted, similarity judgment is made to determinesimilarity to a reference frame image (time A), and image synthesis ismade to produce a single high resolution image from a plurality of frameimages.

Example 4

FIGS. 25 to 27 illustrate examples of classification in a case wherethere is a chronological variation related to the same person.

There may be a case, for example, where images showing a person fromhis/her infancy through childhood to adulthood are to be extracted fromamong a large number of images. However, with current face recognitiontechnologies, making judgments with images shot during a subject'sinfancy and images shot from infancy to adulthood, when the faceundergoes a great change, are difficult; for example, even when a faceduring babyhood has been registered, recognizing a person in images shotin infancy onward as an identical person is difficult. Therefore, whenstill image data 116 is loaded from a digital still camera 112 into apersonal computer (referred to as PC below) 114 as illustrated in FIG.26, and classification into groups is made using an image processingapplication, a message 119 is given on an editing screen 118 that thereis no image showing a face identical to the face of a registered person.

When the message 119 is given although a person represented in the stillimage data 116 loaded in the PC is identical to a registered person, theperson represented in the still image data 116 is newly registered toreconstruct the recognition judgment processing. When a person's face isadditionally registered as illustrated in FIG. 27, faulty judgment doesnot occur and there is no need of additional registration for the timebeing, and still image data 120 of a few images need be onlyoccasionally registered as the person grows. When the features of aperson's face do not change much as after he/she has grown up, there isalso no need of additional registration.

Thus, even when there is a chronological change in the same person,registration of only a few images of face representing each growthperiod suffices to enable classification into groups without thepossibility of faulty judgment.

Example 5

FIGS. 28 and 29 illustrate examples where face recognition is performedwith a still image and frame images, and layout and arrangement ofimages are achieved automatically. Image extraction is performed in sucha manner that, for example, as in the case of the above example 3, mostdesirable images are extracted from frame images extracted from a movingimage and still images.

As illustrated in FIG. 28, when, for example, a photograph album is tobe produced as a present to grandparents of a grandchild, a layout to bemade may be about the grandchild. When a photograph album is to beproduced for a family to view, a layout may be made such that, asillustrated in FIG. 29, the family (father, mother, and a child) andlandscapes are shown evenly. Further, in a photograph album using imagesshot during a trip taken with a friend, a layout for the user may mostlyinclude images showing the user while when a photograph album isproduced as a present for the friend, a layout for the user may mostlyinclude images showing the friend.

Example 6

FIGS. 30 and 31 illustrate examples where character synthesis areas aredetermined using area extraction of a major subject, a sky, and theground. Image extraction is performed in such a manner that similarly tothe above example, most desirable images are extracted from frame imagesextracted from a moving image and still images.

FIG. 30 shows a case where characters are superposed on a marginal areaof a photograph album, i.e., an area containing no image therein. On theother hand, FIG. 31 shows a case where, in the left page of a photographalbum, a vacant region 122 is selected in the layout, and characters aresuperposed on the vacant region 122. On the right page, face regions 124are selected in the layout, and characters are superposed on an areaclear of the face regions 124.

Thus, characters are positioned in such regions as those representingthe skies and the ground or those clear of persons' faces to produce anattractive photograph album.

Example 7

FIG. 32 illustrates an example where a background template is selectedaccording to color information of selected images (major image). Imageextraction is effected in such a manner that, for example, similarly tothe above example 3, most desirable images are extracted from frameimages extracted from a moving image and still images.

FIG. 32 illustrates a case where colors (background color, backgroundtemplate) are designated for marginal portions of a photograph album.When colors are designated for marginal portions of a photograph album,a major image 126 is first extracted from the page. The major imageherein designates an image placed at the center and has a large displaysize.

Next, the major image 126 is analyzed to determine the most used colors.In the example illustrated in FIG. 32, the blue sky in the backgroundrepresents a large area in the major image 126, so that “blue,” forexample, is extracted. The background color (background image) isdetermined according to the extracted color (“blue”). When “blue,” forexample, is extracted, watery blue, which is lighter than “blue” in themajor image is set as the background color so that the border betweenthe marginal area 128 and the blue sky in the major image is clear.Thus, a watery-blue background color is set as background template.

In the above example, a single color background is used but theinvention is not limited this way; a previously provided backgroundimage or an image obtained by processing an image to be placed in thelayout may be used as background image.

Thus, setting background colors (background images) specific to therespective pages produces an impression of unity in image among thepages, enabling production of a more attractive photograph album.

As described above, the image processing device and the image processingmethod of the invention enable production of arrangement data where astill image and optimal frame images are arranged according to theanalysis information and the relationship with a still image withoutrequiring much effort for selecting frame images the user desires to usefrom an image even when the material data (files) contain moving imagedata and still image data mixed therein. Further, the image processingdevice and the image processing method of the invention enableproduction of a print, a photograph album, or an electronic albumaccording to the arrangement data.

The type of the output for producing the print is not limited in anymanner, provided that a print is produced using the present invention,and includes, for example, silver halide photography and ink jetprinting. The print may be produced not only as a sheet of photographbut on such goods as various cards, calendars, mugs, T-shirts, and thelike.

Although the frame images extracted from a moving image have the sameresolution and aspect ratio as the moving image as precondition, alayout may be made by synthesizing a plurality of frame images from amoving image to produce and arrange a panoramic image and producecorresponding arrangement data.

The steps taken in the above image processing method may be configuredinto an image processing program for causing a computer to execute thesteps of the image display method described above, or may be configuredinto an image processing program enabling a computer to function asmeans for executing the respective steps of the image processing methodor to function as means for forming components of the image processingdevice described above.

Further, the above image processing program may be configured in theform of a computer readable medium or a computer readable memory.

While the image processing device, image processing method and programaccording to the invention have been described in detail above, thepresent invention is not limited to the above embodiments, and variousmodifications may be made without departing from the spirit and scope ofthe invention.

I claim:
 1. An image processing device for making a layout of images,comprising: an image data input unit that receives data of moving imagesand still images, an image grouping unit that classifies the movingimages and the still images into groups under a given condition, animage analyzer that analyzes the moving images and the still images asclassified by the group, and that obtains analysis information obtainedby analyzing the moving images and the still images and relationshipinformation indicating relationship between the moving images and thestill images, a frame image extractor that determines a number of frameimages to be extracted from each of the moving images of the groupaccording to a number of still images contained in the group and atleast one of the analysis information and the relationship information,and extracts a determined number of frame images, said still imagescontained in the group not including the frame images that make up themoving images, a layout determining unit that determines a layout of thestill images and the frame images by the group according to at least oneof the analysis information and the relationship information, and animage arranging unit that arranges the still images and the frame imagesby the group according to the layout.
 2. The image processing deviceaccording to claim 1, wherein the given condition is at least one of ashooting time, a shooting location, and a subject.
 3. The imageprocessing device according to claim 1, wherein the relationshipinformation is information comparing the analysis information on themoving images and the analysis information on the still images.
 4. Theimage processing device according to claim 3, wherein the relationshipinformation further comprises representative values calculated byweighting the respective analysis information on the moving images andthe still image, and wherein the representative values are compared tojudge whether there is a relationship between the frame images and thestill image.
 5. The image processing device according to claim 1,wherein the relationship information is the analysis information.
 6. Theimage processing device according to claim 1, wherein the analysisinformation uses at least one of items including face recognition of asubject's face, a number of subjects, a size of a subject, a position ofa subject, a direction in which a subject faces, a direction of asubject's face, recognition of subject's facial expression, a detectionof movement, a shooting time, a shooting location, a composition, avanishing point, colors, brightness of a whole image, vividness of awhole image, identification of a person, a camera movement amount, ashooting condition, a resolution of an image, an aspect ratio of animage, and a shooting mode.
 7. The image processing device according toclaim 6, wherein the layout determining unit determines a layout of thestill images and the frame images according to at least two items of theanalysis information.
 8. The image processing device according to claim6, wherein the frame image extractor extracts the frame images showinghow a face direction changes according to the direction of a subject'sface in the analysis information and the relationship information, andwherein the layout determining unit decides on a layout in which theextracted frame images showing how a face direction changes are arrangedin a given direction.
 9. The image processing device according to claim6, wherein the layout determining unit decides on a layout in whichother still images and frame images are so superposed on the stillimages or the frame images containing a major subject as not to hide themajor subject contained in the still images or the frame imagesaccording to a position of a subject in the analysis information and therelationship information.
 10. The image processing device according toclaim 9, wherein the layout determining unit decides on a layout inwhich the other still images and frame images are not arranged in agiven space located in a direction of a face of the major subject. 11.The image processing device according to claim 6, wherein the imageanalyzer compares a face of a person contained in the still images and aface of a person whose images each representative of growth periods areregistered to identify the person contained in the still images as theperson in the analysis information.
 12. The image processing deviceaccording to claim 1, wherein the image analyzer determines a numberobtained by subtracting a number of still images in a same group from areference number of images, which is a maximum number of images that maybe arranged by the group, as a maximum number of the frame images thatmay be extracted.
 13. The image processing device according to claim 12,wherein the frame image extractor determines the number of frame imagesto be extracted from each of the moving images that does not exceed themaximum number of the frame images that may be extracted, and extractsthe determined number of frame images.
 14. The image processing deviceaccording to claim 1, wherein the layout determining unit decides on alayout in which the frame images are chronologically arranged around thestill images.
 15. The image processing device according to claim 1,wherein layout determining unit decides on a layout in which other stillimages or the frame images are arranged around the still images or theframe images having a depth according to a vanishing point or an imagedirectionality.
 16. The image processing device according to claim 1,wherein the frame image extractor extracts the frame images forexpressing a movement according to the analysis information and therelationship information, and wherein the layout determining unitdecides on a layout in which the extracted frame images arechronologically arranged to express a movement.
 17. The image processingdevice according to claim 1, wherein the frame image extractor extractsthe frame images and produces a frame image by combining a referenceframe image from among the extracted frame images and a frame imagesimilar to the reference frame image.
 18. The image processing deviceaccording to claim 1, wherein the layout comprises a backgroundtemplate, and wherein the layout determining unit determines thebackground template according to color information of the still image orthe frame image being a major image.
 19. The image processing deviceaccording to claim 1, wherein when a recomposition instruction for agiven page is entered by a user, the layout determining unit recomposesa layout in the given page according to the recomposition instruction.20. The image processing device according to claim 1, further comprisinga recorder for recording data of the moving images and the still images,data of the frame images, data of the layout, and data of an arrangementby the image arranging unit, and a display for displaying a result ofthe arrangement.
 21. The image processing device according to claim 20,wherein the groups are a plurality of groups, and extraction results ofthe plurality of groups are simultaneously displayed on the display whenthe frame images are extracted.
 22. The image processing deviceaccording to claim 1, wherein when the moving images have a resolutionlower than a given resolution and a total number of the still images andthe frame images in each of the groups is smaller than a given number,the frame image extractor makes an adjustment by increasing the numberof the frame images to be extracted from the moving images so that adisplay size of one image is smaller than a given size.
 23. The imageprocessing device according to claim 1, wherein when a size of a personcontained in the moving images is smaller than a given size, adjustmentis made by reducing the number of the frame images extracted from themoving images so that the person is displayed in a size not smaller thana given size.
 24. The image processing device according to claim 1,wherein the frame image extractor increases the number of the frameimages to be extracted from the moving image having an aspect ratiorequired for the layout and reduces the number of the frame images to beextracted from other moving images.
 25. The image processing deviceaccording to claim 1, wherein the frame image extractor determines aregion corresponding to a part of a display screen of the moving imagesand extracts the frame images with a determined region.
 26. The imageprocessing device according to claim 1, wherein the frame imageextractor determines the number of the frame images to be extracted fromthe moving images according to at least one of reproduction time of themoving images, an imaged content of the moving images, and relationshipinformation.
 27. The image processing device according to claim 26,wherein when a total number of the still images and the frame images ineach of the groups exceeds a reference number of images, which is amaximum number of images that may be arranged in each of the groups, theframe image extractor extracts recommended frame images from theextracted frame images within a range that does not exceed the referencenumber of images according to the analysis information.
 28. The imageprocessing device according to claim 26, wherein when a total number ofthe still images and the frame images in each of the groups exceeds areference number of images, which is a maximum number of images that maybe arranged in each of the groups, the frame image extractor divides agroup where the total number exceeds the reference number into two ormore groups.
 29. The image processing device according to claim 1,wherein the frame image extractor extracts a number of the frame imagesselected by a user according to a reference number of images, which is amaximum number of images that may be arranged in each of the groups,from the moving images.
 30. The image processing device according toclaim 1, wherein the frame image extractor extracts a number of theframe images selected by a user from the moving images and, when a totalnumber of the still images and the frame images in each of the groupsexceeds a reference number of images, which is a maximum number ofimages that may be arranged in each of the groups, divides a group wherethe total number exceeds the reference number into two or more groups.31. The image processing device according to claim 1, wherein the frameimage extractor determines a number of the frame images to be extractedaccording to a number of pages set by a user.
 32. A print productionsystem comprising: the image processing device described in claim 1, anda printing device for printing data of an arrangement by the imagearranging unit.
 33. A photograph album production system comprising: animage processing device described in claim 1, a printing device forprinting data of an arrangement by the image arranging unit, and and abook binding device for binding sheets of printed paper.
 34. An imageprocessing method of producing a layout of images, comprising: an imagedata entering step of receiving data of moving image and still image, animage grouping step of classifying the moving images and the still imageinto groups under a given condition, an image analyzing step ofanalyzing the moving images and the still images as classified by thegroup, and obtaining analysis information obtained by analyzing themoving images and the still images and relationship informationindicating relationship between the moving images and the still images,a frame image extracting step of determining a number of frame images tobe extracted from each of the moving images of the group according to anumber of still images contained in the group and at least one of theanalysis information and the relationship information, said still imagescontained in the group not including the frame images that make up themoving images, and extracting a determined number of frame images, alayout determining step of determining a layout of the still images andthe frame images by the group according to at least one of the analysisinformation and the relationship information, and an image arrangingstep of arranging the still images and the frame images by the groupaccording to the layout.
 35. A non-transitory computer readable mediumstoring a computer program for causing a computer to perform an imageprocessing method of making a layout of images described in claim 34.